Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance opt pt 1 #1359

Merged
merged 5 commits into from
Oct 6, 2018
Merged

Performance opt pt 1 #1359

merged 5 commits into from
Oct 6, 2018

Conversation

pmconrad
Copy link
Contributor

@pmconrad pmconrad commented Oct 4, 2018

This is some low hanging fruit discovered during my work on #1079 . I'm afraid the fully parallel pre-computation of blocks and transactions will not be finished in time for the upcoming release, therefore submitting this PR first.
The 4th commit in particular help to greatly improve startup time and should also be helpful with get_key_references API call. Related: #1151

@pmconrad pmconrad added 2f Testing Status indicating the solution is being evaluated against the test case(s) 3c Enhancement Classification indicating a change to the functionality of the existing imlementation 6 Performance Impacts flag identifying system/user efficiency, performance, etc. labels Oct 4, 2018
@pmconrad pmconrad added this to the 201810 - Feature Release milestone Oct 4, 2018
Copy link
Contributor

@jmjatlanta jmjatlanta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a manageable-sized bite of performance improvements. I am running a sync on an old db to verify, but the code looks good. I'll update this when the sync finishes.

auto trx_id = trx.id();
FC_ASSERT( (skip & skip_transaction_dupe_check) ||
trx_idx.indices().get<by_trx_id>().find(trx_id) == trx_idx.indices().get<by_trx_id>().end() );
transaction_id_type trx_id;
Copy link
Contributor

@jmjatlanta jmjatlanta Oct 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that you did this. It is more easily readable IMO. Currently, you're protected in all cases with the ifs. But hopefully someone later will know better to not trust that trx_id is set.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will address this in #1360 (id is cached inside transaction there)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid using an uninitialized id, perhaps wrap it with optional? May cause some overheads though.

jmjatlanta
jmjatlanta previously approved these changes Oct 5, 2018
Copy link
Contributor

@jmjatlanta jmjatlanta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran on Ubuntu 18.04 with Boost 1.67, OpenSSL 1.10. Reindexed, replayed, ran without issue. Code looks good.

Copy link
Member

@abitmore abitmore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it's still WIP, perhaps will have answer later. Oops, made in wrong PR.

@@ -106,6 +109,8 @@ void database::reindex( fc::path data_dir )
flush();
ilog( "Done" );
}
if( head_block_time() >= now - gpo.parameters.maximum_time_until_expiration )
skip &= ~skip_transaction_dupe_check;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this is better than the old code to prevent duplicate transactions from being pushed, I think it's not 100% safe due to the use of now(). For example, when maximum_time_until_expiration is 1 day, if last blocks in local database is 1 day before, after finished replay, the dupe-trx db would be empty, so when pushing new blocks received from p2p, dupe check will always pass even if a duplicate transaction is in a block.

Ideally we should use the timestamp of head block after replayed as now, however according to the gap processing below, perhaps it's hard to get head before replay is finished.

Alternatively, we can skip dupe-check during replay, and revisit the pushed blocks to fill the dupe-check db just before enabling undo_db. In order to determine how many blocks to revisit, we can keep track of the greatest maximum_time_until_expiration appeared during replay (assuming it's T) and least block_interval appeared (assuming it's I), then blocks_to_revisit = T/I.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct, but I think it doesn't matter. The dupe check is really only relevant when a witness signs a new block, and that happens only close to now().
If replay ends at now - 1 day, then either we receive blocks created in the past 24 hours and have their tx's added to the dupe database, or there's a gap in the blockchain and the next block will be generated at now() which means older tx's have already expired.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is it's not only about block generation but also about block validation. In case when witnesses colluded signing/accepting blocks with duplicate transactions, other nodes remain the ability to recognize the misbehavior and reject invalid blocks.

I agree practically it's not a big deal. We've been running without that check for years, due to restrict timing/conditions required to exploit the potential issue. E.G. it only affects nodes that just finished a replay, but during normal operations most nodes won't be in that state.

IMHO we can leave the logic as proposed in this PR, but better add some comments there explaining why we did so and what scenarios have been considered and why ignored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After rethinking I switched to last_block->timestamp instead of now. After a chain halt (of course these won't happen again, so it doesn't really matter) it wouldn't require colluding witnesses - any witness could sneak a block with a dupe into the gap.
(Note that this would be noticed during a full sync.)

@pmconrad
Copy link
Contributor Author

pmconrad commented Oct 6, 2018

Is it normal that reviews get dismissed automatically when I push a new commit?

@abitmore
Copy link
Member

abitmore commented Oct 6, 2018

Is it normal that reviews get dismissed automatically when I push a new commit?

Yes. It's configured in repository settings.

@pmconrad pmconrad merged commit fd4a4f2 into develop Oct 6, 2018
@pmconrad pmconrad deleted the performance_opt_1 branch October 6, 2018 18:53
@pmconrad
Copy link
Contributor Author

pmconrad commented Oct 6, 2018

Thanks guys!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2f Testing Status indicating the solution is being evaluated against the test case(s) 3c Enhancement Classification indicating a change to the functionality of the existing imlementation 6 Performance Impacts flag identifying system/user efficiency, performance, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants